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Abstract — This  paper  describes  the  design  and  initial 
positive  evaluation  of  a  prototype  adaptive  automation 
system  to  create  an  enhanced  command  and  control  (C2) 
infrastructure  for  more  effective  operation  of  unmanned 
vehicles.1,2  Our  main  project  objective  is  to  apply  recent 
advances  in  cognitive  engineering  and  display  automation  to 
create  Technology  for  Enhanced  Command  and  Control  of 
Small  Robotic  Assets  (TECRA).  The  initial  goal  is  an 
enhanced  C2  system  for  small  unmanned  aircraft  vehicles 
(SUAVs).  Our  approach  is  to  use  adaptive  display 
technology  to  improve  shared  situation  awareness  between 
the  SUAV  Commander  and  the  SUAV  Operator,  to  provide 
new  channels  of  Commander-Operator  communication,  and 
to  reduce  Commander  workload. 

At  the  core  of  our  approach  is  a  tri-modal  adaptive  interface 
display  which  involves  adaptive  information  presentation  in 
order  to  balance  workload  and  to  promote  effective  human- 
system  performance.  This  novel  design  came  about  as  a 
direct  result  of  field  observations  during  a  full-scale  military 
exercise  and  a  cognitive  task  analysis  (CTA)  based  on  these 
observations.  Using  the  CTA,  we  designed  the  basic 
Commander’s  adaptive  interface  format  and  automated 
triggering  methods.  A  priori  GOMS  analysis  predicted  a 
50%  decrease  in  time  on  task,  based  on  a  subset  of 
representative  tasks.  Data  collection  to  date  supports  these 
predictions.  Furthermore,  feedback  from  subject  matter 
experts  and  comparisons  between  user  performance  on 
TECRA  versus  an  existing  SUAV  platform  suggests  that 
TECRA  is  easier  to  use,  quicker  to  learn,  and  provides  more 
capabilities  to  the  user  than  current  systems. 

These  results  demonstrate  how  the  TECRA  application  - 
driven  by  a  cognitive  analysis  of  the  Commander’s  task,  by  a 
mission  model  of  the  anticipated  Commander’s  needs,  and 
by  mission  templates  and  real-time  robotic  data  -  has  been 
able  to  validate  theories  of  human-automation  interaction  in 
real-world  domains  such  as  unmanned  aviation  and  military 
command  and  control. 
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1.  Introduction 

Military  forces  of  the  future  will  use  mixed  manned  and 
unmanned  forces  for  a  broad  variety  of  functions: 
reconnaissance  and  surveillance,  logistics  and  support, 
communications,  forward-deployed  offensive  operations, 
and  as  tactical  decoys  to  conceal  maneuvers  by  manned 
assets.  Mandates  to  reduce  manning  in  the  military  have  led 
to  initiatives  to  assign  multiple  heterogeneous  unmanned 
vehicles  to  a  small  number  of  human  team  members.  The 
goal  of  such  robot-human  teams  is  to  extend  manned  and 
unmanned  capabilities  and  act  as  “force  multipliers”,  as  in 
the  US  Army  Future  Combat  System  [1,2,3].  Robot-human 
teams  introduce  a  new  and  unique  aspect  to  the  planning, 
coordination  and  evaluation  of  unit  performance. 

Among  the  most  successful  fielded  unmanned  systems  are 
the  Small  Robotic  Assets  (SRAs),  both  Small  Unmanned  Air 
Vehicles  (SUAVs)  and  Small  Unmanned  Ground  Vehicles 
(SUGVs).  Recent  successes  in  Iraq  have  provided  an 
indication  of  their  potential  for  revolutionizing  the  way  U.S. 
troops  conduct  operations.  SUAV  scout  planes  and  sensor 
systems  have  made  it  easier  to  spot  insurgents  and  roadside 
bombs,  thus  saving  lives. 

As  a  result,  the  U.S.  Army  is  committing  increased  resources 
to  developing  enhanced  surveillance,  communications  and 
weapons  for  SUAVs.  Off-the-shelf  SUAVs  such  as  the 
fixed  wing  Raven  are  currently  deployed  in  sizable  numbers. 
By  March  2006,  for  example,  the  Raven  had  been  deployed 
on  more  than  15,000  sorties  or  deployments  totaling  18,673 
flight  hours.  With  regard  to  new  platform  technology,  Army 
spokesman  LTC  John  Kelleher  has  said,  “We  are  going  to 
compare  what  is  out  there  -  the  commercial,  off-the-shelf, 
fixed  wing  assets,  such  as  Raven  -  with  the  ducted-fan 
technology  that  the  Army  is  developing,  and  we  will  make  a 
decision  on  which  way  we  are  going  to  go.” 

There  are  many  essential  human  factors  that  need  to  be 
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optimized  for  the  SUAV  operators  themselves,  and  there  is 
much  research  being  done  to  address  these  issues. 
Neumann  and  Durlach  [4]  provide  a  brief  review  of  factors 
affecting  a  user’s  ability  to  teleoperate  a  robot  or  vehicle, 
and  many  other  publications  cover  this  ground  also.  The 
presently  proposed  paper,  however,  focuses  on  the  equally 
important  but  relatively  neglected  problem  of  coordination 
of  SUAV  operator  and  unit  commander  team  performance. 
It  is  important  to  assure  that  commander-operator  team 
performance  is  optimized  with  respect  to  such  key  factors  as 
efficient  communication,  teamwork,  well-distributed 
workload  and  effective  operation  of  the  SUAV  system  in 
order  to  realize  the  full  potential  contribution  of  the  SUAV. 
This  is  particularly  true  as  manning  considerations  make  it 
more  likely  that  one  unit  commander  will  control  a  number 
of  small  robotic  assets,  thus  further  complicating  the  team 
interactions.  Accordingly,  there  is  a  pressing  need  for 
analysis  of  and  technological  solutions  to  the  problems 
currently  preventing  effective  command  and  control  of  small 
robotic  assets. 

The  most  critical  problem  areas  in  the  command  and  control 
of  SUAV  assets  include: 

(1)  Inadequate  information  flow  from  the  SUAV  to 
the  small  unit  Commander.  The  Commander  has 
insufficient  information  on  current  and  planned 
SUAV  operations,  and  inadequate  contextual 
knowledge  with  which  to  interpret  and  guide  the 
progress  of  the  overall  mission. 

(2)  Poor  coordination  between  Commander  and 
SUAV  operator(s).  There  is  typically  little  or  no 
common  view  of  the  SUAV’  activities  and 
capabilities,  a  limited  voice  communications 
channel,  and  consequently  a  high  frequency  of 
miscommunication  regarding  the  long  term  and 
immediate  mission  objectives  and  tasks. 

(3)  Limited  commander  training  in  terminology  and 
technical  details  of  the  vehicle  operation.  This 
lack  of  knowledge  exacerbates  communication 
problem  and  contributes  to  system  ineffectiveness 
and/or  failure. 

(4)  Inadequate  systematic  methods  for  training  and 
feedback.  This  includes  no  standard  after  action 
review  (AAR)  methodology  for  commander- 
operator  coordination,  no  standardized  team 
performance  measures  and  no  evaluation  metrics  to 
inform  systematic  training. 

This  paper  describes  the  design  and  initial  investigation  of  a 
prototype  command  and  control  system  aimed  at  creating  an 
enhanced  C2  infrastructure  that  allows  more  effective 
operation  of  unmanned  vehicles.  To  demonstrate  and  test 
this  prototype,  we  decided  to  use  small  unmanned  aircraft 
vehicles  (SUAV)  as  our  initial  target  platform  because  of 


their  immediate  importance  to  current  military  operations, 
and  because  analysis  has  shown  that  the  current  absence  of 
an  adequate  command  infrastructure  is  a  key  detriment  to 
their  fully  effective  utilization.  In  addition,  we  selected  the 
Raven  SUAV  as  our  specific  use  case  and  evaluation 
testbed. 

We  describe  the  Raven  SUAV  platform,  followed  by  a 
detailed  description  of  our  design  and  prototype  system.  We 
follow  this  with  a  summary  of  the  interviews  that  we 
conducted  with  subject  matter  experts  (SME)  who  reviewed 
our  system.  We  conclude  with  a  description  of  an 
experiment  aimed  at  validating  our  design  prototype  and 
testing  the  efficacy  of  the  interface. 


2.  The  Raven  SUAV  System 

The  Raven  is  a  compact,  lightweight  SUAV  that  can  be 
prepared  and  hand-launched  in  minutes  for  the  purpose  of 
conducting  aerial  intelligence,  surveillance,  and 
reconnaissance  (ISR)  missions  during  infantry  combat 
operations,  including  urban  warfare  operations/MOUT.  The 
basic  Raven  package  consists  of  the  battery  powered  air 
vehicle  (AV)  and  a  set  of  video  camera  payloads,  a  ground 
control  unit  (GCU)  and  a  remote  video  terminal  (RVT) 
which  is  essentially  a  GCU  without  uplink  capability  [5,6]. 
Figure  1  shows  the  normative  Raven  C2  configuration  for 
the  most  recent  RQ-  11B+  Version.  The  components  are  as 
follows: 

Vehicle  Operation — The  SUAV  is  normally  operated  by  a 
2-person  team  of  Vehicle  Operator  (VO)  and  Mission 
Operator  (MO)  who  are  located  close  together.  The  SUAV 
is  flown  directly  by  the  VO  using  the  GCU,  which  displays 
the  direct  video  view  from  the  SUAV  as  well  as  flight 
information  such  as  coordinates.  Mission  planning  in  terms 
of  waypoints,  modes,  etc.,  as  well  as  mission  monitoring, 
adjustment  and  review  is  done  by  the  MO  on  a  dedicated  PC 
using  the  specialized  Talon  Tool  for  the  general  Falcon  View 
map  display  program. 

Data  Transmission — Flight  control  data  from  the  VO  and 
MO  stations  are  transmitted  by  radio  to  the  SUAV.  These 
data  include  the  mission  waypoints,  mission  modes,  etc.,  as 
well  as  direct  flight  commands,  and  are  stored  on-board  the 
SUAV.  The  SUAV  re-transmits  the  stored  mission  data 
along  with  real-time  updates  as  flight  status  data  and  also 
transmits  real-time  video  information.  The  data  transmitted 
from  the  SUAV  are  displayed  in  various  formats  on  the 
GCU,  the  Talon  Tool  enabled  FalconView  PC  and  the  RVT. 

Command — The  Commander  under  whom  the  Raven  is 
being  flown  is  typically  located  remotely  from  the  VO/MO 
operating  team.  He  monitors  the  mission  on  his  RVT,  which 
displays  the  real-time  video  signal  and  a  superimposed  set  of 
flight  status  information.  He  may  also  use  a  PC  with 
FalconView  maps  for  improved  situation  awareness,  but  the 
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FalconView  application  will  not  include  Raven  real-time 
flight  status  information. 

Communication — The  Commander  communicates  with  the 
MO  and/or  VO  using  voice  over  a  radio  net.  He  may 
communicate  directly  or  through  an  intermediary,  generally 
the  fire  control  officer.  The  VO  and  MO  normally 
communicate  with  each  other  directly  as  they  are  usually 
close  enough  together,  but  also  share  the  radio 
communication  net. 


3.  Adaptive  Automation  Design 
Methodology 

Our  challenge  was  to  design  an  adaptive  interface  for 
commanders  that  facilitated  information  flow  between  the 
SUAV  and  the  humans  and  enhanced  commander-vehicle 
operator  coordination.  A  further  challenge  was  to  design 
our  new  interface  so  that  it  could  work  together  with  the 
current  Raven  SUAV  system.  In  the  current  Raven  system, 
the  commander  primarily  has  a  RVT  that  serves  as  the  main 
source  of  visual  imagery  from  the  SUAV.  The  RVT  display 
is  identical  to  that  of  the  VO,  and  consists  of  several  flight 
parameters  superimposed  over  a  video  scene  (see  Figure  2). 

Our  approach  was  to  1)  enhance  the  current  commander’s 
display  by  adaptively  displaying  relevant  information  as  the 
mission  progressed,  and  2)  extend  the  current  capabilities  of 
the  system  by  turning  the  display  into  a  portal  through  which 


the  commander  could  access  advanced  features  and 
communicate  intent  to  the  Raven  operators.  We  used  our 
own  adaptive  design  methodology  that  consisted  of  five 
steps. 

Step  1:  Field  Observations 

Extensive  field  observations  were  conducted  at  two 
locations  (Ft.  Benning,  GA  and  Ft.  Polk,  LA)  in  order  to 
understand  the  commanders’  tasks  and  their  information 
needs. 

At  Ft.  Benning,  the  observations  focused  on  Army  doctrine, 
training,  and  schoolhouse  knowledge  of  small  UAVs  by 
observing  a  day  of  training  at  the  Small  Unmanned  Aircraft 
System  (SUAS)  Course.  During  this  day,  current  Raven 
operators  were  being  trained  to  become  Raven  instructors 
for  their  units.  At  Ft.  Polk,  we  observed  company 
commanders  and  their  infantry  units  over  several  days  of  a 
week-long  full-scale  exercise  intended  to  prepare  Army  units 
for  upcoming  deployments. 

The  observational  data  collected  from  both  locations 
resulted  in  a  wealth  of  information  including  50  hours  of 
field  observations,  2  hours  of  audio  transcriptions  and  over  3 
hours  of  videotape  footage  from  interviews  with  Raven 
operators,  a  copy  of  the  current  Raven  manual,  and 
classroom  presentation  materials  from  the  SUAS  course. 


Raven  SUAV 
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Figure  1  -  Normative  Raven  Command  and  Control  setup 
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Step  2:  Cognitive  Task  Analysis 

Using  the  knowledge  gained  from  the  schoolhouse  doctrine 
and  manuals,  we  developed  a  comprehensive  Hierarchical 
Task  Analysis  (HTA)  that  highlighted  the  typical  tasks  a 
commander  performs  during  a  Raven  mission  as  well  as  the 
specific  information  requirements  associated  with  those 
tasks.  This  task  analysis  was  then  used  to  prepare  probes  for 
the  field  observations  made  during  the  training  exercise.  The 
results  from  this  second  trip  served  to  inform  a  modified 
version  of  Klein,  Calderwood,  and  MacGregor’s  [7]  Critical 
Decision  Method  Cognitive  Task  Analysis  (CTA). 

The  HTA  and  the  CTA  revealed  an  interesting  pattern.  The 
commander’s  tasks  could  be  sorted  into  three  general  task 
categories:  monitoring  the  current  video  feed,  reviewing  past 


information,  and  re-tasking  the  vehicle  in-flight.  We  refer  to 
these  categories  collectively  as  mission  phases. 

In  addition  to  this  emergent  categorization  of  tasks,  the  CTA 
helped  identify  several  bottlenecks  that  might  plague  a 
typical  mission.  The  most  common  (and  arguably  the  most 
disruptive)  bottleneck  was  the  radio  communications 
between  commander  and  Raven  operators,  as  shown  by  the 
operational  sequence  diagram  (see  Figure  3).  Factors 
contributing  to  this  bottleneck  were  radio  frequency 
congestion,  intermediaries,  and  the  absence  of  the  message 
recipient. 

A  second  bottleneck  stemmed  from  the  commander’s 
incomplete  understanding  of  the  Raven  system,  which 
sometimes  leads  to  additional  radio  communications.  A 
final  bottleneck  was  the  slow  manner  in  which  information 
obtained  by  the  Raven  was  distributed  from  the  command 
post  to  the  rest  of  the  company. 

Step  3:  Preliminary  Adaptive  Interface  Design:  Overview 

Our  newly  proposed  design  was  a  departure  from  the  current 
system  in  which  the  commander  must  make  do  with  raw 
flight  data  and  spend  precious  time  waiting  for  queries,  re¬ 
tasking  directives  and  responses  to  be  relayed  over  the  radio 
net  to  the  vehicle  operators.  Using  the  bottlenecks  listed 
above  as  critical  areas  of  improvement,  we  identified  three 
main  system  requirements  to  guide  our  design.  First,  the 
new  interface  must  reduce  reliance  on  verbal  radio 
communications.  Second,  the  interface  must  be  intuitive. 


Figure  3  -  Operation  Sequence  Diagram  showing  communications  bottleneck  for  commander 
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And  third,  the  interface  must  facilitate  the  production  of 
products  for  later  consumption,  such  as  during  patrols  and 
After  Action  Reviews  (AARs). 

In  short,  we  envisioned  an  interactive  display  that  bypasses 
the  brittle  and  cumbersome  nature  of  voice  radio 
communications  by  creating  a  data  link  between  the 
interfaces  of  the  commander  and  the  operators. 

Mission  Modes 

We  proposed  three  analogous  Mission  Mode  displays  based 
on  the  three  distinct  mission  phases  identified  by  the  CTA. 
These  give  the  commander  the  flexibility  he  needs  to 
manage  the  UAV  asset  effectively  based  on  his  information 
needs  and  time  available.  The  modes  are  briefly  described 
below  and  displayed  in  Figure  4. 

Monitor  Mission  Mode  Display — The  Monitor  view  is  the 
default  view  and  will  allow  the  commander  to  watch  the 
Raven  video  feed  in  real-time.  The  video  display  is  largest 
in  the  configuration  to  allow  easy  situation  awareness 
assessment  by  the  commander.  The  current  location  of  the 
asset  is  readily  available  in  a  thumbnail  map  while  details  of 
what  is  being  tracked  appear  in  the  mission  analysis  panel. 

Review  Mission  Mode  Display — The  Review  Mission  view 
allows  the  commander  to  examine  stored  imagery  of  targets 
and  landmarks  that  have  been  obtained  by  the  Raven.  In  this 
mode  the  commander  can  flip  through  captured  images,  and 
video  clips,  obtain  distances  for  roads  and  landmarks,  read 
target-specific  information  entered  by  the  MO,  and  mark  up 
the  images  with  several  annotation  tools  in  the  Mission 
Analysis  panel. 

Change  Mission  Mode  Display — The  change  mission  mode 
allows  the  commander  to  quickly  signal  a  course  change  to 
the  Raven  team  when  needed.  The  map  is  largest  in  this 
configuration  to  allow  accurate  flight  plan  review  and  in¬ 
flight  re-tasking  requests. 


1 .  Monitor  Mission  2.  Review  Mission  3.  Change  Mission 


Figure  4  -  The  three  modes:  Monitor,  Review,  and 
Change  Mission  Mode 

Step  4:  Selecting  Invocation  Methods 

The  proposed  interface  will  switch  modes  adaptively  based 


on  (1)  mission  type,  (2)  critical  events,  and  (3)  individual 
preferences  of  the  commander.  The  critical  question  is: 
Which  invocation  points  should  trigger  the  adaptive  mode 
selection  and  when  should  they  be  activated? 

Our  approach  to  this  issue  was  to  use  the  goals  and  methods 
from  the  HTA  to  build  a  GOMS  model  for  the  commander. 
The  methods  in  this  model  describe  the  specific  goals  a 
commander  can  accomplish  with  the  interface.  The  decisions 
represent  the  choices  the  commander  has  to  make  when 
accomplishing  these  goals.  The  selection  rules  describe 
which  procedural  IF-THEN  rule  the  commander  can  follow 
when  multiple  options  are  available.  The  model  provides  a 
useful  framework  to  support  the  various  invocation  methods. 

Mission  Type — Different  missions  require  different  goals  to 
be  accomplished.  For  instance,  reconnaissance  missions 
typically  have  a  specific  aim  in  mind  such  as  “verify  target 
X  is  at  location  Y”  while  surveillance  involves  a  more 
general  goal  of  observing  whatever  can  be  seen.  Depending 
on  the  mission  at  hand,  mission  specific  goals  can  be 
included  or  excluded  from  the  model. 

Critical  Events — During  a  mission,  certain  events  may 
require  an  interface  adaptation.  For  example,  upon  spotting 
a  suspicious  car,  an  operator  engages  the  loiter  mode  and  the 
left  camera  for  the  Raven.  Following  such  a  sequence,  the 
commander’s  display  could  switch  to  the  Monitor  Mode: 

Selection  Rule  for  Goal:  MONITOR  MODE 

IF  Flight  mode  =  Loiter  and  Camera  =  Left 
and  Monitor  Mode  =  off, 

THEN  Accomplish  Goal:  SWITCH  TO  MONITOR 

MODE 

IF  Flight  mode  =  Loiter  and  Camera  =  Left 
and  Monitor  Mode  =  on, 

THEN  Accomplish  Goal:  MAINTAIN  MONITOR  MODE 

Commander  Preference — The  system  can  also  learn  the 
preferences  of  commanders  by  monitoring  and  analyzing 
their  goals  and  decisions  and  incorporating  such  information 
into  a  Bayesian  network.  For  example,  one  commander  may 
frequently  take  a  picture  of  the  same  object  of  interest.  The 
system  may  offer  to  take  the  picture  for  commanders  if  they 
have  a  high  probability  of  deciding  to  do  so. 

Step  5:  Implement  and  Validate  Method  of  Invocation 

One  of  the  well-documented  benefits  of  GOMS  is  the  ability 
to  make  a  priori  predictions  about  performance  times  of 
expert  behavior.  Thus,  we  tested  the  feasibility  of  our 
design  in  the  very  early  stages  of  development  by  comparing 
the  predicted  performance  times  of  our  proposed  interface 
against  the  predicted  performance  times  of  the  current 
system  for  three  representative  tasks.  The  results  showed  a 
48%  improvement  in  performance  time,  shaving  almost  5 
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minutes  off  a  nearly  10  minute  series  of  tasks.  Based  on 
these  very  promising  results,  we  could  justify  continued 
development  of  the  new  design. 


4.  The  TECRA  Prototype 

The  TECRA  prototype  was  designed  to  create  a  new 
command  and  control  infrastructure  that  will  support  the 
Commander  and  improve  Commander-Operator  team 
performance.  Another  goal  was  to  enhance  Commander- 
Operator  coordination.  Based  on  analyses  described  in  the 
previous  section,  we  developed  an  integrated  Commander’s 
display  that  combines  video,  navigational,  and  mission 
information.  The  TECRA  system’s  Commander  display 
makes  use  of  existing,  as  well  as  new  mission  information 
data  links  to  facilitate  shared  situation  awareness  among  the 
Commander  and  the  SUAV  Operators  throughout  the 
mission.  This  section  describes  in  detail  the  TECRA 
functionality  and  interface  design. 

TECRA-Raven  Configuration 

The  projected  new  TECRA-Raven  configuration  is  shown  in 
Figure  5.  The  most  important  elements  are  described  below: 

Additional  flight  data  sent  to  TECRA  Commander 
interface — The  flight  status  data  can  be  provided  to  the 
Commander’s  PC  via  an  already  available  connection  from 


the  RVT  unit.  This  data  incorporates  mission  planning  and 
route  data  entered  by  the  MO,  representing  almost  all  of  the 
information  available  on  the  MO’s  PC,  as  well  as  real  time 
SUAV  data.  This  new  information  will  be  supplied  to  the 
Commander  via  the  new  TECRA  Commander  Interface. 

New  MO-CMDR  data  link  will  enhance  communication  and 
shared  awareness — In  addition,  a  new  radio-based  data  link 
is  established  between  the  Commander’s  and  the  MO’s  PC, 
which  will  enable  (1)  direct  texting  between  the  two  team 
members,  to  improve  mission-related  communication,  and 
(2)  transmission  to  the  Commander’s  PC  of  MO  inputs  not 
included  in  the  SUAV  downlink  signals  to  improve  shared 
awareness  by  coordinating  the  two  displays. 

TECRA  Commander ’s  Interface 

A  completely  new  interface  was  designed  and  developed  for 
the  mission  Commander  (see  Figure  6).  This  interface 
consists  of  the  tri-modal  adaptive  component  display.  The 
three  mission  modes  correspond  to  specific  mission  tasks 
that  Commanders  typically  engage  in.  The  specific  TECRA 
interface  components  are  described  below: 

Area  Map — The  area  map  (upper  left)  is  synchronized  with 
the  vehicle  operator’s  FalconView,  giving  a  common  picture 
of  the  authorized  airspace,  planned  route,  and  areas  of 
interest.  Commanders  can  draw  directly  on  the  map  for  their 
own  use,  print  off  copies  for  dissemination,  or  send  the 
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Figure  5  -  Projected  TECRA-Raven  C2  Configuration 
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markup  to  the  operators  for  a  clear  explanation  of  their 
intent.  Commanders  can  also  create  suggested  waypoints  or 
routes  and  submit  them  to  the  vehicle  operators  for  review. 

Video  Feed — The  video  feed  (upper  right)  comes  directly 
from  the  UAV  and  presents  selected  flight  data. 

Commanders  can  freeze  the  frame,  rewind,  fast-forward, 
take  screenshot  pictures,  and  even  record  video  clips,  while 
the  mission  is  underway.  The  field  of  view  is  depicted  on  the 
map,  not  just  showing  the  commander  “Where  am  I?”  but 
also  “What  am  I  looking  at?” 

Mission  Timeline — The  mission  timeline  (lower  left)  is  able 
to  track  multiple  vehicles  and  record  events  such  as 
waypoints,  when  photos  and  video  clips  were  taken,  and 
user-created  bookmarks  for  later  review. 

Saved  Media  Library — The  saved  media  library  (middle 
left)  gives  easy  access  to  screen  captures  and  video  clips, 
which  are  sorted  and  listed  in  a  library  of  locally  stored 
pictures  and  videos.  Each  item  is  tied  to  a  geographical 
location  which  appears  on  the  map  when  the  image  is 
reviewed. 

Message  Center — The  message  center  (lower  right)  lets 


Commanders  communicate  with  their  operators  through  a 
built-in  chat  feature  that  enables  users  to  send/receive 
queries  and  updates,  as  well  as  coordinates  and  mark-up 
symbols. 

Toolbar — The  toolbar  (far  right)  permits  editing  of  both 
maps  and  images,  with  built-in  drawing  tools.  In  addition  to 
these  tools,  the  Commander  can  also  save  and  export  new 
images  to  external  applications. 


5.  SME  Evaluation 

The  first  step  in  evaluating  our  new  TECRA  interface  was  to 
have  military  commanders  use  it  and  evaluate  its  efficacy. 
We  recruited  officers  at  Ft.  Campbell  to  participate  in  this 
study.  Results  of  this  study  are  described  in  this  section. 

Method 

We  had  a  sample  size  of  six,  including  three  Captains,  one 
Lieutenant,  and  two  enlisted  service  members  from  1st  and 
3rd  BCTs. 

Our  sessions  ran  about  2  hours.  We  began  with  an  interview 
to  gauge  the  background  of  the  participant,  and  then 
introduced  the  TECRA  system.  Participants  learned  the 


Figure  6  -  TECRA  Commander  Interface 
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basics  of  the  system  via  one-on-one  training,  and  then  were 
asked  to  perform  the  similar  tasks  on  their  own. 
All  audio  was  recorded,  as  were  keystrokes  and  screenshots. 
When  time  was  tight,  this  training  session  was  reduced  to  a 
demonstration,  and  feedback  was  elicited  from  the 
participant  during  the  demo.  Following  the  TECRA  session, 
we  administered  another  questionnaire  that  focused  on  the 
interface. 

Results 

The  responses  to  TECRA  were  overwhelmingly  positive. 
The  commanders  we  spoke  with  confirmed  that  our  system 
adds  functionality  they  currently  do  not  possess,  as  did  the 
Raven  operators.  In  addition,  commanders  rated  the  system 
as  user-friendly  and  easy  to  learn.  Some  of  their  actual 
comments  included: 

(1)  “It’s  a  pretty  simple  system.  I  would  use  this.” 

(2)  “I  think  this  is  useful  in  a  laptop  version  or  being 
able  to  interface  on  a  website  based  thing...  on  a 
computer  in  a  company  CPU  would  be  very 
useful.” 

(3)  “Yeah,  I  mean  it  all  seems  pretty  user  friendly, 
especially  if  you  sat  down  and  played  with  it  a  few 
times.  It’s  pretty  quick  to  figure  out.” 

(4)  “I  think  this  is  a  good  little  system.” 


6.  Experimental  Study 

Our  next  goal  was  to  evaluate  the  TECRA  system  more 
formally.  In  particular,  we  were  interested  in  comparing  the 
current  remote  viewing  terminal  (RVT)  system  to  our  new 
TECRA  system  and  determining  if  we  would  find  any 
performance  benefits.  We  decided  to  simply  compare  the 
currently  used  RVT  interface  with  the  newly  designed 
TECRA  interface. 

Several  measures  were  used  in  this  experiment  to  compare 
the  two  interfaces.  Besides  performance  on  tasks,  we  were 
particularly  interested  in  two  important  human  performance 
issues  including  maintaining  or  improving  situation 
awareness  and  balanced  mental  workload.  One  major  system 
design  goal  is  to  facilitate  high  levels  of  situation  awareness 
for  all  team  members  involved.  This  is  a  major  concern  for 
UV  operators,  but  it  is  also  important  for  Commanders  or 
other  individuals  consuming  sensor  data  from  the  UV.  By 
increasing  the  situational  awareness  of  UV  customers,  the 
overall  mission  should  become  more  efficient. 

When  shared  situational  awareness  is  not  employed, 
inefficient  human-robot  teaming  may  result.  Situation 
awareness  issues  such  as  unreliable  voice  communication 


between  a  commander  and  vehicle  operator  or  a  lack  of  a 
common  view  of  the  asset’s  activities  between  the 
commander  and  the  operator  may  cause  unnecessarily  high 
cognitive  demands,  in  an  already  highly  cognitive 
demanding  and  stressful  environment. 

TECRA  is  designed  to  make  it  easy  for  users  to  find 
information  on  the  screen  and  in  particular,  to  help  inform 
users  where  they  are  geographically  and  what  they  are 
looking  at.  Theoretically,  this  design  should  improve 
situation  awareness  when  they  are  using  the  system.  In 
addition,  subjectively  experienced  workload  should  be 
reduced  because  the  display  was  designed  to  be  sparse  only 
showing  those  elements  that  are  useful  to  the  operator, 
therefore  hypothetically  reducing  the  effort  needed  to  locate 
critical  information. 

We  therefore  predicted  that  users  of  the  TECRA  software 
would  have  an  easier  time  locating  information  and  thus 
show  an  increase  in  situation  awareness  for  the  TECRA 
system  compared  to  the  current  RVT  display.  We  also 
predicted  a  decrease  in  subjectively  experienced  workload. 
In  addition,  we  expected  that  users  would  perform  better  on 
target  detection  tasks  compared  to  the  RVT. 

Method 

Twelve  students  from  George  Mason  University  (4  Males 
and  8  Females)  participated  in  this  study  and  were 
compensated  with  course  credit.  After  signing  an  informed 
consent  form,  the  experimenter  read  the  experimental 
instructions  to  the  participant.  Participants  were  also  shown 
picture  examples  of  the  targets  that  they  were  instructed  to 
identify.  Participants  did  not  receive  any  training  on  either 
of  the  two  SUAV  monitoring  interfaces.  This  was  done  so 
we  could  examine  how  easy  it  was  to  learn  to  use  either  of 
these  systems. 

Target  Detection  Task — Participants  were  asked  to  monitor 
a  video  feed  from  an  SUAV  using  the  new  prototype 
TECRA  interface  and  the  currently  used  RVT  interface.  The 
task  was  to  identify  target  buildings.  The  majority  of  the 
scenery  displayed  on  the  video  feed  was  two  dimensional 
graphics.  Targets  consisted  of  3-dimentional  buildings, 
which  would  appear  intermittently  throughout  each  scenario. 
Participants  were  instructed  to  respond  to  each  target  by 
pressing  a  button  and  briefly  describing  the  size  and  color  of 
the  building. 

Situation  Awareness  Task — In  addition  to  the  target 
detection  task,  participants  were  also  instructed  to 
simultaneously  answer  questions  sent  to  them  via  instant 
“chat”  messages.  Messages  were  sent  once  every  35-45 
seconds.  Participants  were  asked  to  respond  to  each  message 
by  typing  and  sending  their  answer  via  the  chat  window. 
Two  types  of  questions  were  asked:  secondary  distracter 
questions  (simple  arithmetic  problems);  and  Situation 
Awareness  (SA)  questions.  The  SA  questions  concerned  the 


current  state  of  the  SUAV.  These  questions  included: 

(1)  What  waypoint  is  the  vehicle  heading  toward? 

(2)  What  is  the  ground  speed  of  the  vehicle? 

(3)  What  is  the  altitude  of  the  vehicle? 

(4)  What  direction  is  the  vehicle  traveling  (cardinal 
direction  or  heading)? 

These  specific  questions  were  chosen  as  measures  of  SA  for 
two  reasons:  1)  each  of  these  four  characteristics  (upcoming 
waypoint,  vehicle  speed,  vehicle  altitude,  and  vehicle 
direction)  were  identified  as  important  pieces  of  information 
for  the  Commander,  especially  when  making  a  route  request, 
and  2)  both  the  TECRA  interface  and  the  RVT  interface 
display  this  information,  therefore  making  it  possible  for  the 
participant  to  answer  all  SA  questions  using  either  of  these 
interfaces. 

In  both  conditions  (TECRA  interface  and  RVT  interface)  all 
of  the  information  needed  to  answer  the  SA  questions  was 
included  on  the  video  feed  overlay.  However,  the  two 
interfaces  displayed  and  organized  the  information 
differently. 

Participants  completed  four  trials,  two  trials  using  the 
current  RVT  system  and  two  trials  using  the  TECRA 
prototype.  Each  trial  lasted  approximately  10  minutes  and 
consisted  of  12-14  targets  and  8-11  situational  awareness 
questions. 

The  conditions  were  counterbalanced  to  account  for 
ordering  effects.  Participants  also  completed  the  NASA- 
TLX  workload  index  after  each  trial. 

Results 

All  data  was  analyzed  using  a  2x2  Repeated  Measures 
ANOVA  with  condition  Interface  Type  (RVT,  TECRA)  and 
Scenario  (1,2).  Results  described  include  data  for  target 
detection  accuracy  and  reaction  time,  situation  awareness 
and  subjective  workload. 

Target  Detection — No  significant  differences  were  found  for 
target  detection  accuracy,  F(l,ll)  =  0.238,  p  >  0.05,  and 
reaction  time,  F(l,ll)  =  1.29,  p  >  0.05. 


Figure  7  -  Target  Detection  Accuracy 


However,  there  was  a  trend  in  favor  of  the  TECRA  system. 
Accuracy  was  slightly  higher  in  the  TECRA  interface 
condition  (M  =  69%,  SEM  =  4%)  compared  to  the  RVT 
condition  (M  =  68%,  SEM  =  4%)  (see  Figure  7).  Reaction 
time  was  also  slightly  faster  on  average  by  1  second  in  the 
TECRA  condition  (see  Figure  8).  Low  power  due  to  the 
small  sample  was  probably  the  reason  why  this  result  was 
not  significant. 


Figure  8  -  Target  Detection  Reaction  Time  (seconds) 


Situation  Awareness — There  was  a  significant  effect  for 
Interface  Type  for  situation  awareness  accuracy,  F(  1,11)  = 
44.57,  p  <  0.05,  and  situation  awareness  reaction  time, 
F(l,ll)  =  15.58,  p  <  0.05.  Situation  awareness  accuracy  was 
markedly  higher  in  the  TECRA  condition  (M  =  92%,  SEM  = 
7%)  compared  to  the  RVT  condition  (M  =  38%,  SEM  =  6%) 
(see  Figure  9).  Furthermore,  SA  reaction  time  was  faster  in 
the  TECRA  condition  (M  =  5.9s,  SEM  =  0.9s)  compared  to 
the  RVT  condition  (M  =  9.2s,  SEM  =  0.9s)  (see  Figure  10). 
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Figure  9  -  Situation  Awareness  Accuracy 


Figure  10  -  Situation  Awareness  Reaction  Time 
(seconds) 


Subjective  Workload — There  was  a  significant  effect  for 
Interface  Type  for  subjective  workload,  F(l,ll)  =  6.70,  p  < 
0.05.  Subjective  workload  was  lower  in  the  TECRA 
condition  (M  =  41.7,  SEM  =  3.0)  compared  to  the  RVT 
condition  (M  =  45.2,  SEM  =  3.4)  (see  Figure  11).  For  both 
systems  workload  was  relatively  low  at  an  average  of  43.5. 


Figure  11  -  Subjective  Workload  (NASA-TLX) 


Discussion 

The  goal  of  this  experiment  was  to  compare  the  TECRA 
interface  to  the  RVT  interface  in  terms  of  performance  and 


user  experience.  We  found  that  users  of  the  TECRA 
interface  showed  an  increase  in  situation  awareness,  a 
reduction  in  situation  awareness  reaction  time,  and  a 
reduction  in  subjective  workload  compared  to  the  RVT 
system.  Furthermore,  trends  showed  that  the  TECRA  system 
may  improve  target  detection  performance  compared  to  the 
RVT  system  although  these  results  are  not  reliable. 

The  results  empirically  demonstrate  the  efficacy  of  the 
TECRA  interface.  Improvements  in  situation  awareness  are 
vital  as  they  can  matter  in  time-critical  situations  where  the 
timely  and  accurate  information  can  save  lives.  In  addition, 
reducing  subjective  workload  was  one  of  the  major 
objectives  of  this  program  and  cited  as  a  critical  requirement 
for  the  commander. 

The  results  are  also  consistent  with  the  original  GOMS 
model  that  predicted  a  reduction  in  time  needed  to  operate 
the  TECRA  system  compared  to  the  original  RVT  system. 
Furthermore,  this  experiment  only  consisted  of  a  basic 
monitoring  task.  The  systems  were  not  used  for  making 
route  requests,  taking  pictures  or  video  clips  from  the  live 
video  feed,  or  relaying  and  receiving  critical  information 
from  the  MO.  It  is  likely  that  the  observed  performance 
improvements  when  using  TECRA  for  conducting  a  simple 
monitoring  task  would  extent  to  benefiting  a  Commander 
when  conducting  additional,  more  complex  C2  tasks. 
Further  research  will  need  to  be  conducted  to  measure  the 
benefits  of  other  TECRA  functionality. 


7.  Conclusions 

Our  goal  for  this  project  was  to  design  a  system  that  would 
improve  coordination  and  performance  of  SUAS  crews.  We 
did  this  in  a  systematic  manner.  First  we  analyzed  the 
current  situation  by  interviewing  users  of  the  Raven  system 
and  observing  military  exercises.  We  then  developed  our 
own  design  methodology  to  design  an  adaptive  automation 
interface  system  based  on  previous  theory  [1].  We  took  a 
prototype  of  the  interface  and  showed  it  to  SMEs  who 
provided  us  with  feedback.  Finally,  we  conducted  an 
experiment  to  examine  whether  our  system  would  improve 
overall  performance. 

The  SME  interviews  showed  that  the  TECRA  system  was 
easy  to  use,  easy  to  learn,  and  added  functionality  that  did 
not  yet  exist.  The  experiment  showed  that  situation 
awareness  improved  and  subjective  workload  was  reduced 
when  using  the  TECRA  system  in  comparison  to  the  RVT 
system.  Taken  together,  these  are  encouraging  results  in 
favor  of  the  TECRA  system. 

Several  future  research  directions  are  being  planned.  First, 
we  intend  to  further  test  the  adaptive  features  of  the  interface 
in  a  follow-up  experiment  with  a  larger  sample  size.  We 
plan  to  compare  several  conditions  including  a  situation 
where  there  is  no  automation,  non-adaptive  (static) 
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automation,  or  user-centered  adaptive  automation.  We  also 
plan  to  conduct  field  tests  to  test  the  TECRA  system  in  a 
real-world  setting  with  a  real  Raven  SUAV. 

With  further  testing  our  TECRA  system  can  improve  in  its 
design  and  may  eventually  be  incorporated  into  existing 
SUAV  systems. 
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